Skip to content

feat(testing): add DeepSeek live qualification, Steward v2.4.0 eval cases, and pytest speedups - #9201

Merged
loopx-agent merged 1 commit into
mainfrom
codex/deepseek-behavior-and-test-opt-20261121
Nov 21, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/deepseek-behavior-and-test-opt-20261121

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Summary

Adds opt-in live DeepSeek model-behavior & Steward evaluation support plus vision-aligned pytest speedups without mutating shared production prompt/runtime contracts (rules.py, task_body.py, chat.py, isolate_sqlite_runtime) or docs/product/releases/v2.4.0.md.

Part A — DeepSeek Live Behavior Qualification & Steward Evaluation Coverage

  • loopx/control_plane/testing/doubao_model_behavior_actor.py: Keeps all DOUBAO_*, _direct_ark_transport, and Doubao*Actor classes intact while adding DeepSeek constants (DEEPSEEK_FLASH_MODEL="deepseek-flash", DEEPSEEK_V4_PRO_MODEL="deepseek-v4-pro", ALLOWED_DEEPSEEK_BEHAVIOR_MODELS), resolve_deepseek_chat_completions_endpoint (canonical https://api.deepseek.com/chat/completions, failing closed with noncanonical_endpoint on non-canonical URLs), _direct_deepseek_transport (_NoRedirectHandler), _strip_json_markdown_fence, DeepSeekModelBehaviorActor, and DeepSeekOnboardingModelBehaviorActor (actor_ref="deepseek:<model>", default deepseek-flash). Zero unowned Literal/Enum types introduced.
  • loopx/control_plane/testing/model_tool_behavior.py + 5 tool-behavior actors: Parameterizes DoubaoExecToolClient with optional endpoint, allowed_models, and provider_prefix, adds DeepSeekExecToolClient, and adds optional client=None + from_deepseek_environment(...) to all 5 tool-behavior actors (selected_todo, replan_semantic_action, scoped_gate_successor, capability_monitor_repair, terminal_settlement) reusing existing state machines and semantic oracles.
  • scripts/qualify-deepseek-model-behavior-live.py, scripts/qualify-host-prompt-release.py, host_prompt_behavior.py: Adds scripts/qualify-deepseek-model-behavior-live.py (--max-workers 6 default), extends scripts/qualify-host-prompt-release.py with --provider {ark,deepseek} and --model, passes synthetic registry_path=Path("registry.json") in host_prompt_behavior.py::probe_messages to prevent dual global-root lookup crashes on multi-host machines, and strips optional JSON markdown fences in run_probe.
  • examples/evaluations/chat-intake.py, chat-intake.public.json, docs/product/use-cases/steward/golden-queries.md: Adds bounded --workers (1..12, default 6) and 2 public-safe Steward cases (unconfirmed-pr-merge-status-only, watch-only-monitor-not-completed; 28 → 30 cases), plus DeepSeek unit coverage in tests/control_plane/test_actual_default_model_behavior_live.py, tests/control_plane/test_host_prompt_behavior.py, and tests/test_chat_intake_evaluation.py.
  • Docs updated: docs/reference/protocols/model-behavior-qualification-v0.md, docs/development/testing-and-quality.md, docs/product/release-readiness.md, docs/product/use-cases/steward/golden-queries.md (docs/product/releases/v2.4.0.md intentionally left untouched).

Part B — Vision-Aligned Pytest Speedups (Zero Invariant Weakening)

  1. tests/conftest.py::_isolated_worker_effect_runtime: Sets scoped 0o700 NODE_COMPILE_CACHE inside each worker's isolated temporary runtime unless NODE_DISABLE_COMPILE_CACHE=1 or NODE_V8_COVERAGE is set.
  2. tests/control_plane/canonical_authority_fixture.py: Replaces repeated per-call Node cold-starts in initialize_canonical_authority / advance_canonical_fixture_authority with a thread-locked persistent Popen Node worker per (pid, package_root, sqlite_enabled) with atexit cleanup and 1x auto-respawn while keeping isolate_sqlite_runtime unchanged (test_canonical_authority_fixture.py: 7 + 2 new tests = 9 passed).
  3. tests/control_plane/test_canonical_lease_scopes.py: Routes clean default CLI calls in-process via loopx.cli.main (redirect_stdout), retaining subprocess.run whenever cwd or LOOPX_LEASE_SCOPE_* is overridden.
  4. benchmark/tests/test_task_entry.py::test_execute_wake_keeps_host_clocks_out_of_fresh_and_resumed_input: Replaces 16x full Cartesian product with a 6-row pairwise covering matrix preserving the [900, 500] clock loop.
  5. tests/control_plane/test_dependency_vision_wait_cli.py: Reduces wait/restart polling iterations (100, 200) → (12, 24) (len(sleeps) == 24 and sleeps[-1] == 3600), still crossing 3-poll limit + saturated 3600s ticks across restart.

Verification

Exact Head (624a2a2a6579e7f149ea02b7965ab257406b9bc7 on origin/main 0a0c2a7c8b)

  • Static, docs, vocabulary, boundary:
    • ruff check on all 20 touched Python files: passed (0 errors)
    • mypy: Success: no issues found in 17 source files
    • scripts/check_rfc_bilingual.py --base origin/main --head HEAD: passed
    • examples/release/release-readiness-doc-smoke.py: passed
    • loopx check across all 12 touched directory/file roots (795 files): ok: True, errors=0
    • Fast Part A & fixture tests (7 files): 132 passed in 5.38s
  • Goal-bound CQR (personal-loopx-maintenance): cqr_9216b09eb905ec64d47f (status: valid, decision: pass, blocking: false, scope 9216b09eb905ec64d47fbf39d601e968b5071ecaf82ec9059d8d43a04d76d882)
  • loopx canary premerge --from-git-diff --git-diff-base origin/main --goal-id personal-loopx-maintenance --timeout-seconds 300: 4/4 direct checks passed, 1/1 py_compile passed, 19/19 selected checks passed (10/10 catalog canaries incl. semantic-vocabulary-drift-smoke.py + control-plane-maintainability-ratchet-smoke.py, 8/8 risk profile smokes, 1/1 public boundary), 0 failures, quality_receipt_failure_count: 0, manual_hold_count: 1 (benchmark_sensitive on benchmark/tests/test_task_entry.py)

Additional Pre-Rebase / Exact-Code Suite & Timing Evidence (same PR diff)

  • Full targeted Part A + fixture suites (14 files on 3688ec61cb): 384 passed (132 in 6.07s + 121 in 194.20s + 131 in 234.20s, 0 failed, 0 skipped)
  • Part B Before vs After wall timings (same 4 files, 0 skipped/relaxed assertions):
    • tests/control_plane/test_canonical_authority_fixture.py: 2.80s (7 passed) → 1.73s (7 passed, quiet) / 5.69s (9 passed incl. 2 new worker tests on 3688ec61cb)
    • tests/control_plane/test_dependency_vision_wait_cli.py: 110.76s (20 passed) → 40.44s (quiet) / 69.61s (20 passed under concurrent release lanes on 3688ec61cb)
    • tests/control_plane/test_canonical_lease_scopes.py: 302.79s (194 passed) → 113.46s (quiet) / 134.40s (194 passed under load on 3688ec61cb)
    • benchmark/tests/test_task_entry.py -k test_execute_wake_keeps_host_clocks_out_of_fresh_and_resumed_input: 408.23s (16 passed) → 152.69s (quiet) / 161.27s (6 passed under load on 3688ec61cb)
    • Combined 4-suite pytest total: 824.58s → 308.32s (-62.6%, quiet) / 370.97s (-55.0%, under concurrent load)

Live DeepSeek Qualification & Steward Evals (Honest Unfiltered Baseline)

  • 22-scenario actual-default portfolio (qualify-deepseek-model-behavior-live.py --max-workers 6):
    • deepseek-flash: 22/22 scenarios & 6/6 contrasts passed (qualification_passed=True) in 97 calls (3688ec61cb), 100 calls (379fa74955), and 101 calls (9911d1162c); on 624a2a2a65 scored 21/22 scenarios & 6/6 contrasts (128 calls; repeat 1 passed in 11 calls, repeat 2 of turn_required_vision_replan hit tool_call_budget_exhausted at 40/40 calls with shell_nonzero: 2).
    • deepseek-v4-pro (3688ec61cb): 17/22 scenarios & 6/6 contrasts passed (133 calls; failed: onboarding_agent_identity_gate, turn_terminal_settlement, turn_planning_horizon_strategic_context, turn_required_vision_replan, turn_scoped_gate_successor_replan).
  • qualify-host-prompt-release.py --release-live --provider deepseek --repeats 2:
    • deepseek-flash: 17/20 passed (4/4 quiet_work, quiet_wait, vision_replan, wait_fallback_work; failed only notifying_wait: thin x2, brief attempt 2).
    • deepseek-v4-pro: 16/20 passed (4/4 on those 4 cases; failed notifying_wait on thin x2, brief x2).
    • Diagnostic finding (kept shared rules.py/task_body.py untouched per scope): both models interpreted quota_skip + agent_channel.delivery_allowed=false + "Output only under user_channel.notify=NOTIFY; otherwise stay quiet" as needing no user output even when operator_question and NOTIFY are present.
  • Steward live evaluations (examples/evaluations/chat-intake.py --live --provider operator-api --workers 6 --repeats 2):
    • deepseek-flash: chat-intake 48/60 (unconfirmed-pr-merge-status-only 2/2, watch-only-monitor-not-completed 1/2), chat-routing 5/18, chat-followthrough 7/14, chat-material 6/14, chat-purpose 1/8.
    • deepseek-v4-pro: chat-intake 47/60 (unconfirmed-pr-merge-status-only 2/2), chat-routing 6/18, chat-followthrough 5/14, chat-material 10/14, chat-purpose 4/8.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google · self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

在精确 head 624a2a2a6579e7f149ea02b7965ab257406b9bc7 上完整复核了 PR #9201 全部 25 个改动文件,并独立运行了核心单测、子进程回退等价性验证、正交成对覆盖数学校验、静态检查与预合并门禁:DeepSeek 实网测试入口与双全局根修复严谨,5 处测试加速全部保持原始语义且未触碰共享提示词规则与 v2.4.0.md。

动机 / Motivation

发版维护者与开发者在执行 DeepSeek 实网行为核查、Steward Golden Query 评估以及本地全量 pytest 回归测试时,会受到工具覆盖缺失与测试冷启动耗时过长的影响。改动前,行为资格套件仅支持豆包 Ark 端点,host_prompt_behavior 在同时存在双全局根目录时会触发探测异常且无法剥离 Markdown JSON 代码块,Steward 评估仅覆盖 28 条用例且默认 2 并发,本地测例则因每次夹具冷启 Node、子进程调用 CLI 及过宽的笛卡尔乘积与轮询次数耗时较久;改动后新增严格的 DeepSeek 端点与模型白名单校验、双根安全传参、30 条公开评测与默认 6 并发,并通过可自动重启的持久化 Node Worker、进程内 CLI 快速路径、隔离编译缓存及正交成对覆盖显著缩短测试耗时且未弱化断言。实测表明,共享提示词与控制面核心文件(loopx/control_plane/quota/、loopx/chat.py、loopx/heartbeat_prompt.py)及 docs/product/releases/v2.4.0.md 均为 0 行改动;DeepSeek 非规范 URL、缺失密钥、脏仓及未传 --release-live 均严格快速失败;Steward 30 条用例与 --workers 1..12 范围校验全部通过;持久化 Node Worker 自动重拉、in-process 与 LOOPX_LEASE_SCOPE 外部子进程双路径等价性、6 行成对矩阵 24/24 两两交互覆盖以及 12/24 轮询重启边界均已验证通过。本 PR 不修改核心控制面运行时语义与共享提示词契约(rules.py、task_body.py、chat.py、isolate_sqlite_runtime),不改动 docs/product/releases/v2.4.0.md,也不要求在无密钥的常规 CI 中调用外部付费模型接口。

改动思路 / Approach

DeepSeek 适配层通过参数化复用了现有直接对话/工具调用基类、传输异常分类及 5 套行为状态机(未产生平行控制面决策或私有枚举),同时所有 pytest 加速均严格限定在 tests/ 与 benchmark/tests/ 夹具边界内,核心控制面代码(quota/、chat.py、heartbeat_prompt.py)与 v2.4.0 发布说明保持零修改。当前 PR 边界严格限定在测试评估套件、实网质检脚本、测试夹具加速及对应开发/协议文档,不修改共享提示词规则、核心调度状态机或稳定版发布说明。

flowchart TD
    A["DeepSeek / Eval 入口 (624a2a2a65)"] --> B{"--live / --release-live & 干净仓 & Key?"}
    B -- "缺参或脏仓" --> B1["FAIL-CLOSED: skip(0 调用) 或 RuntimeError"]
    B -- "非规范 BASE_URL" --> B2["noncanonical_endpoint 拒绝请求"]
    B -- "通过" --> C["复用 _DirectChat* / DeepSeekExecToolClient / 5 状态机"]
    D["Pytest 加速路径"] --> E["_CanonicalAuthorityWorker 按 (pid,pkg,sqlite) 复用与 1 次自动重拉"]
    D --> F["test_canonical_lease_scopes 默认 cli_main,有 cwd/env 时回退子进程"]
    D --> G["6 行成对矩阵 (24/24 两两覆盖) & 12/24 轮询 (达 3600s)"]
Loading

基于精确 head 624a2a2a6579e7f149ea02b7965ab257406b9bc7 源码与本地检查整理,仅作控制流示意。

核查面 基线 (0a0c2a7c8b) 当前 head (624a2a2a65) 独立验证结论
端点/密钥/脏仓安全门 仅支持 Ark,无 DeepSeek 端点规范 resolve_deepseek_chat_completions_endpoint 仅放行官方根路径//v1 PASS (非规范 URL/缺密钥/脏仓均拦截)
双全局根与围栏 JSON probe_messages 未传 registry_path 显式传入合成 Path('registry.json') + _strip_json_markdown_fence PASS (test_host_prompt_behavior.py 通过)
Steward 评估集与并发 28 条用例,固定 max_workers=2 30 条用例,--workers 支持 1..12(默认 6) PASS (test_chat_intake_evaluation.py 通过)
持久化 Node Worker 每次初始化/推进均冷启 Node 进程级带锁 _CanonicalAuthorityWorker(TS 依赖 0 process.env) PASS (91 passed in 4.35s + 自动重拉)
CLI / 正交 / 轮询加速 始终子进程 / 16 组笛卡尔 / (100,200) 轮 默认 cli_main + LOOPX_LEASE_SCOPE_* 回退;6 行 24/24 覆盖;(12,24) PASS (子进程对比 2/2、成对 6/6、轮询 2/2)

具体改动 / Changes

本次基于公共分叉基线 0a0c2a7c8bb38bad503bb3eba47cb4a3f0974b4d 共改动 25 个文件(+1593 / -205 行):实网评估与测试适配器(+757 / -127)、测试加速与新增单测(+772 / -72)及 4 份文档更新(+64 / -6);确认 docs/product/releases/v2.4.0.md、loopx/control_plane/quota/、loopx/chat.py、loopx/heartbeat_prompt.py 均为 0 行改动。

对照协议规范 docs/reference/protocols/model-behavior-qualification-v0.md(精确修订 624a2a2a6579e7f149ea02b7965ab257406b9bc7):

  • DS-QUAL-1(Direct DeepSeek 对话与 5 类工具行为角色严格限定官方端点、禁用 thinking、模型白名单并复用共享状态机):implemented(通过 test_actual_default_model_behavior_live.py 及脏仓/缺密钥反事实验证)。
  • DS-QUAL-2(Host prompt 发布探针支持 --provider deepseek、默认关闭、剥离 JSON 围栏且免疫双全局根冲突):implemented(通过 test_host_prompt_behavior.py 与默认跳过反事实验证)。
  • STEWARD-EVAL-1(Steward 公开集扩展至 30 条含 PR 状态与监控只读区分用例,并支持 --workers 1..12):implemented(通过 test_chat_intake_evaluation.py 验证)。
  • PYTEST-SPEED-1(测试加速保持 worker/路径隔离、子进程回退、全量两两交互覆盖与 3600s 退避重启回归不变):implemented(通过 7 套单测、子进程等价跑测、24/24 成对校验及 12/24 轮询验证)。

关键代码讲解

  1. loopx/control_plane/testing/doubao_model_behavior_actor.py — resolve_deepseek_chat_completions_endpoint(L89-101)与 _direct_chat_completions_transport(L104-169):规范化 DEEPSEEK_BASE_URL 仅接受官方根 URL 与 /v1,其它地址以 noncanonical_endpoint 阻断;抽取公共 HTTPS 传输路径禁用重定向并统一脱敏 401 鉴权报错,避免暴露密钥。
  2. loopx/control_plane/testing/doubao_model_behavior_actor.py — _strip_json_markdown_fence(L456-472):仅剥离外层空标签或 json 围栏代码块并同时供 _provider_decision 与 host_prompt_behavior.run_probe 复用,对内部非法 JSON 仍保留严格解码拦截。
  3. loopx/control_plane/testing/model_tool_behavior.py — DeepSeekExecToolClient(L343-389)及 5 处 from_deepseek_environment:通过可选 endpoint/allowed_models/provider_prefix 参数复用 DoubaoExecToolClient,令 selected_todo、replan_semantic_action、scoped_gate_successor、capability_monitor_repair 与 terminal_settlement 共用同一状态机。
  4. loopx/control_plane/testing/host_prompt_behavior.py — probe_messages(L59-86):向 build_heartbeat_prompt 显式传递合成 registry_path=Path("registry.json"),阻断在同时存在 ~/.loopx 与 ~/.codex/loopx 全局目录时触发隐式多根歧义报错。
  5. tests/control_plane/canonical_authority_fixture.py — _CanonicalAuthorityWorker(L31-197):按 (pid, package, sqlite_enabled) 维护带线程锁与 45s 选择器超时的持久化 Node 进程,注册 atexit 回收并支持异常退出后 1 次自动重拉;已核查其加载的 4 个 TypeScript 依赖模块无任何 process.env / os.tmpdir 读取。

对主干的风险 / Risks

重点风险核查结论如下:

  • 夹具持久化状态污染风险:已核查 local_authority_provider.ts、file_authority_store.ts、legacy_writer_fence.ts、authority_store_codec.ts 均不读取环境变量或系统临时目录,路径全部来自单次 JSON 请求显式传入的 req.root 与 req.goal;tests/conftest.py 仅在每个 pytest worker 独立临时目录创建 0o700 编译缓存并服从关闭标志;test_canonical_lease_scopes.py 在 cwd 或 LOOPX_LEASE_SCOPE_* 存在时继续走 subprocess.run。
  • 测试缩减覆盖风险:经脚本逐 pair 核算,benchmark/tests/test_task_entry.py 的 6 行参数矩阵覆盖了全部 24/24 个两两因素组合;对照 scripts/external_scheduler_worker.py 的 select_interval,12/24 轮询仍越过重启门槛与最大 3600s 退避台阶。
  • 其他评审维度:scope_fit(verified)所有新增脚本、参数与夹具均有活跃调用点;default_off_isolation / authority_semantics / semantic_alignment 为 not_applicable(无新增未注册控制面枚举,外部请求须显式 --live/--release-live + 密钥触发);Steward 与测试默认值变化均已同步披露文档。

独立本地验证结果:

  • Part A + 权威夹具单测(test_actual_default_model_behavior_live.py、test_host_prompt_behavior.py、test_chat_intake_evaluation.py、test_canonical_authority_fixture.py):91 passed in 4.35s(0 跳过)(PASS)
  • 7 文件合并套件与重点回归:352 passed, 54 skipped(跳过均位于 test_task_entry.py 不相关平台/live 项);6 个非跳过文件收集核对 305 条全过;LOOPX_LEASE_SCOPE_PARITY=1 子进程对照 2 passed in 2.54s;test_execute_wake 6 行矩阵 6 passed in 155.81s;12/24 依赖等待测试 2 passed in 12.61s (PASS)
  • 静态与边界门禁:ruff(20 文件)、mypy(17 文件)、check_rfc_bilingual.py、release-readiness-doc-smoke.py、semantic-vocabulary-drift-smoke.py(29/29 词表)、loopx check(8 根路径 / 1562 文件 0 错,15 条既存非字面量告警)、共享控制面/说明零改动核对:全部通过 (PASS)
  • loopx canary premerge --from-git-diff --git-diff-base origin/main --timeout-seconds 300:选中 19/19 项检查全部通过(failures=0;manual_hold_count=1 来自修改 benchmark/tests/test_task_entry.py 触发的 benchmark_sensitive 提示)(PASS)
  • 脏仓拒绝(rc=1)、缺 DEEPSEEK_API_KEY(rc=1)、未传 --release-live 默认跳过(status='skipped', provider_call_count=0)、24/24 成对矩阵反事实:全部通过 (PASS)
  • 实网付费 DeepSeek API 请求:评审期间未主动消耗外部额度重跑实网,作者在 PR 描述已如实记录通过数及 notifying_wait / Steward 现网基线差距 (UNVERIFIED);远程 GitHub CI 依 Goal 配置未等待 (UNVERIFIED)。

我的整体评价 / Verdict

基于独立校验,PR #9201 在严格保持共享控制面与提示词规则零改动、且隔离测试安全的前提下,完整补齐了 DeepSeek 行为验证与 Steward v2.4.0 公开评估能力,并显著加速了慢速 pytest 套件。problem_context.outcome_impact 评估:long_horizon 为 improved(进程级带锁持久化 Node Worker、严格端点与凭证门禁、跨重启 3600s 退避测试均保持稳健);user_experience 为 improved(维护者获得统一的 --provider deepseek 门禁与双全局根修复且保留显式串行/子进程回退开关);visual_review 为 not_applicable:已复核全部 25 个改动文件,均属于评估脚本、测试框架夹具、行为模拟模块与协议/测试文档,无前端视图或交互式 UI 页面变动;未验证范围(unverified_scope):none。

observable_semantics 判定为 intentional_change_validated(chat-intake --workers 6 / 30 用例、6 行 24/24 成对矩阵与 (12,24) 轮询调整均有测试与文档披露并保留显式回退路径),scope_coverage 判定为 verified,code_volume 为 necessary(兼容性 retain),change_proportionality 为 proportionate 且机制评估 retain,default_off_isolation / authority_semantics / semantic_alignment 均为 not_applicable(无新控制面状态词表,29/29 词表检查通过)。后续可非阻断地抽离 5 个工具行为类中重复的 from_deepseek_environment 模板;注意合并时 benchmark/tests/test_task_entry.py 触发 benchmark_sensitive 提示由授权维护者核实。精确 head 624a2a2a6579e7f149ea02b7965ab257406b9bc7 质量达标,建议批准。

English verdict: APPROVE on exact head 624a2a2: shared prompt/runtime contracts and v2.4.0 release notes have 0 diff lines, DeepSeek endpoints/actors/dual-root/fence fixes and test speedups pass all local suites and counterfactuals (91 Part A+fixture tests PASS in 4.35s, 305 targeted non-skipped tests PASS, LOOPX_LEASE_SCOPE_PARITY subprocess parity PASS, 24/24 pairwise coverage PASS, semantic-vocabulary-drift-smoke PASS, canary premerge 19/19 PASS with manual_hold_count=1 for benchmark_sensitive test matrix reduction); live paid DeepSeek API calls remain un-re-run in review.

@loopx-agent
loopx-agent merged commit d772190 into main Nov 21, 2026
14 of 19 checks passed
@loopx-agent
loopx-agent deleted the codex/deepseek-behavior-and-test-opt-20261121 branch November 21, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant